10 research outputs found
Evaluating Spiking Neural Network On Neuromorphic Platform For Human Activity Recognition
Energy efficiency and low latency are crucial requirements for designing
wearable AI-empowered human activity recognition systems, due to the hard
constraints of battery operations and closed-loop feedback. While neural
network models have been extensively compressed to match the stringent edge
requirements, spiking neural networks and event-based sensing are recently
emerging as promising solutions to further improve performance due to their
inherent energy efficiency and capacity to process spatiotemporal data in very
low latency. This work aims to evaluate the effectiveness of spiking neural
networks on neuromorphic processors in human activity recognition for wearable
applications. The case of workout recognition with wrist-worn wearable motion
sensors is used as a study. A multi-threshold delta modulation approach is
utilized for encoding the input sensor data into spike trains to move the
pipeline into the event-based approach. The spikes trains are then fed to a
spiking neural network with direct-event training, and the trained model is
deployed on the research neuromorphic platform from Intel, Loihi, to evaluate
energy and latency efficiency. Test results show that the spike-based workouts
recognition system can achieve a comparable accuracy (87.5\%) comparable to the
popular milliwatt RISC-V bases multi-core processor GAP8 with a traditional
neural network ( 88.1\%) while achieving two times better energy-delay product
(0.66 \si{\micro\joule\second} vs. 1.32 \si{\micro\joule\second})
Fully Automatic Gym Exercises Recording: An IoT Solution
In recent years, working out in the gym has gotten increasingly more
data-focused and many gym enthusiasts are recording their exercises to have a
better overview of their historical gym activities and to make a better
exercise plan for the future. As a side effect, this recording process has led
to a lot of time spent painstakingly operating these apps by plugging in used
types of equipment and repetitions. This project aims to automate this process
using an Internet of Things (IoT) approach. Specifically, beacons with embedded
ultra-low-power inertial measurement units (IMUs) are attached to the types of
equipment to recognize the usage and transmit the information to gym-goers and
managers. We have created a small ecosystem composed of beacons, a gateway,
smartwatches, android/iPhone applications, a firebase cloud server, and a
dashboard, all communicating over a mixture of Bluetooth and Wifi to distribute
collected data from machines to users and gym managers in a compact and
meaningful way. The system we have implemented is a working prototype of a
bigger end goal and is supposed to initialize progress toward a smarter, more
efficient, and still privacy-respect gym environment in the future. A
small-scale real-life test shows 94.6\% accuracy in user gym session recording,
which can reach up to 100\% easily with a more suitable assembling of the
beacons. This promising result shows the potential of a fully automatic
exercise recording system, which enables comprehensive monitoring and analysis
of the exercise sessions and frees the user from manual recording. The
estimated battery life of the beacon is 400 days with a 210 mAh coin battery.
We also discussed the shortcoming of the current demonstration system and the
future work for a reliable and ready-to-deploy automatic gym workout recording
system
ERICA: Enabling real-time mistake detection and corrective feedback for free-weights exercises
National Research Foundation (NRF) Singapore under International Research Centres in Singapore Funding Initiativ
ColibriES: A Milliwatts RISC-V Based Embedded System Leveraging Neuromorphic and Neural Networks Hardware Accelerators for Low-Latency Closed-loop Control Applications
End-to-end event-based computation has the potential to push the envelope in
latency and energy efficiency for edge AI applications. Unfortunately,
event-based sensors (e.g., DVS cameras) and neuromorphic spike-based processors
(e.g., Loihi) have been designed in a decoupled fashion, thereby missing major
streamlining opportunities. This paper presents ColibriES, the first-ever
neuromorphic hardware embedded system platform with dedicated event-sensor
interfaces and full processing pipelines. ColibriES includes event and frame
interfaces and data processing, aiming at efficient and long-life embedded
systems in edge scenarios. ColibriES is based on the Kraken system-on-chip and
contains a heterogeneous parallel ultra-low power (PULP) processor, frame-based
and event-based camera interfaces, and two hardware accelerators for the
computation of both event-based spiking neural networks and frame-based ternary
convolutional neural networks. This paper explores and accurately evaluates the
performance of event data processing on the example of gesture recognition on
ColibriES, as the first step of full-system evaluation. In our experiments, we
demonstrate a chip energy consumption of 7.7 \si{\milli\joule} and latency of
164.5 \si{\milli\second} of each inference with the DVS Gesture event data set
as an example for closed-loop data processing, showcasing the potential of
ColibriES for battery-powered applications such as wearable devices and UAVs
that require low-latency closed-loop control
ColibriUAV: An Ultra-Fast, Energy-Efficient Neuromorphic Edge Processing UAV-Platform with Event-Based and Frame-Based Cameras
The interest in dynamic vision sensor (DVS)-powered unmanned aerial vehicles
(UAV) is raising, especially due to the microsecond-level reaction time of the
bio-inspired event sensor, which increases robustness and reduces latency of
the perception tasks compared to a RGB camera. This work presents ColibriUAV, a
UAV platform with both frame-based and event-based cameras interfaces for
efficient perception and near-sensor processing. The proposed platform is
designed around Kraken, a novel low-power RISC-V System on Chip with two
hardware accelerators targeting spiking neural networks and deep ternary neural
networks.Kraken is capable of efficiently processing both event data from a DVS
camera and frame data from an RGB camera. A key feature of Kraken is its
integrated, dedicated interface with a DVS camera. This paper benchmarks the
end-to-end latency and power efficiency of the neuromorphic and event-based UAV
subsystem, demonstrating state-of-the-art event data with a throughput of 7200
frames of events per second and a power consumption of 10.7 \si{\milli\watt},
which is over 6.6 times faster and a hundred times less power-consuming than
the widely-used data reading approach through the USB interface. The overall
sensing and processing power consumption is below 50 mW, achieving latency in
the milliseconds range, making the platform suitable for low-latency autonomous
nano-drones as well
Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO
Smart glasses are rapidly gaining advanced functionality thanks to
cutting-edge computing technologies, accelerated hardware architectures, and
tiny AI algorithms. Integrating AI into smart glasses featuring a small form
factor and limited battery capacity is still challenging when targeting
full-day usage for a satisfactory user experience. This paper illustrates the
design and implementation of tiny machine-learning algorithms exploiting novel
low-power processors to enable prolonged continuous operation in smart glasses.
We explore the energy- and latency-efficient of smart glasses in the case of
real-time object detection. To this goal, we designed a smart glasses prototype
as a research platform featuring two microcontrollers, including a novel
milliwatt-power RISC-V parallel processor with a hardware accelerator for
visual AI, and a Bluetooth low-power module for communication. The smart
glasses integrate power cycling mechanisms, including image and audio sensing
interfaces. Furthermore, we developed a family of novel tiny deep-learning
models based on YOLO with sub-million parameters customized for
microcontroller-based inference dubbed TinyissimoYOLO v1.3, v5, and v8, aiming
at benchmarking object detection with smart glasses for energy and latency.
Evaluations on the prototype of the smart glasses demonstrate TinyissimoYOLO's
17ms inference latency and 1.59mJ energy consumption per inference while
ensuring acceptable detection accuracy. Further evaluation reveals an
end-to-end latency from image capturing to the algorithm's prediction of 56ms
or equivalently 18 fps, with a total power consumption of 62.9mW, equivalent to
a 9.3 hours of continuous run time on a 154mAh battery. These results
outperform MCUNet (TinyNAS+TinyEngine), which runs a simpler task (image
classification) at just 7.3 fps per second
Human Activity Recognition with Field Sensing Technique
The development of machine learning algorithms and novel sensing modalities has boosted the exploration of human activity recognition(HAR) in recent years. In this work, we explored field-based sensing solutions and different machine learning models for HAR tasks to address the shortcomings of existing HAR sensing solutions, like the weak robustness of RF-based solution, environment-dependency of the optic-based solution, etc., aiming to supply a competitive and alternative sensing approach for HAR tasks.
Field, in physics, describes a region in which each point will be affected by force. Field sensing is potentially a low-cost, low-power, non-intrusive, privacy-respecting HAR solution that is ideal for long-term, wearable activity recording. By directly/indirectly monitoring the field strength or other field variation caused variables, some unsolved HAR problems could be addressed when other sensing solutions fail. An example is the social distance monitoring problem, where the most widely adopted approach is based on the Bluetooth signal strength measurement. However, the signal is so subtle that any object surrounding the signal emitter will cause signal attenuation. To guarantee the accuracy of social distance monitoring, we developed an induced magnetic field-based social distance monitoring system with an accuracy of a sub-ten centimetre. Moreover, the system is robust and resistant to environmental variations. Like Bluetooth, other RF-wave-based sensing modalities also face the multi-path effect caused by refraction. Thus their signal is unreliable for positioning applications where higher accuracy and robustness are needed. Besides the magnetic field, we also explored a natural static passive electric field, the field between the human body and surroundings, namely the human body capacitance(HBC). HBC is a physiological parameter describing the charge distribution difference between the body and the surroundings and is seldomly explored before. We developed a few wearable, low-cost, low power consumption hardware platforms, either based on an oscillating unit or discrete components composed sensing front end followed by a high resolution analog-to-digital module, to
monitor the variation of the parameter regarding the body movement and environmental variations. Compared with the inertial sensors, the HBC could deliver full-body movement perceiving, meaning that the movement of the legs could be perceived by a wrist-worn HBC sensing unit, which is far beyond the
sensing ability of an inertial sensing unit.
To summarize, we introduced two competitive field sensing modalities for HAR tasks, the magnetic field sensing for position-related services and the passive electric field sensing for full-body action and environmental variation sensing. Both of which were still in an infant stage and not fully explored in the community. The advantages of the two field sensing modalities were demonstrated with a series of position-related and motion-related experiments
Human Activity Recognition with Field Sensing Technique
The development of machine learning algorithms and novel sensing modalities has boosted the exploration of human activity recognition(HAR) in recent years. In this work, we explored field-based sensing solutions and different machine learning models for HAR tasks to address the shortcomings of existing HAR sensing solutions, like the weak robustness of RF-based solution, environment-dependency of the optic-based solution, etc., aiming to supply a competitive and alternative sensing approach for HAR tasks.
Field, in physics, describes a region in which each point will be affected by force. Field sensing is potentially a low-cost, low-power, non-intrusive, privacy-respecting HAR solution that is ideal for long-term, wearable activity recording. By directly/indirectly monitoring the field strength or other field variation caused variables, some unsolved HAR problems could be addressed when other sensing solutions fail. An example is the social distance monitoring problem, where the most widely adopted approach is based on the Bluetooth signal strength measurement. However, the signal is so subtle that any object surrounding the signal emitter will cause signal attenuation. To guarantee the accuracy of social distance monitoring, we developed an induced magnetic field-based social distance monitoring system with an accuracy of a sub-ten centimetre. Moreover, the system is robust and resistant to environmental variations. Like Bluetooth, other RF-wave-based sensing modalities also face the multi-path effect caused by refraction. Thus their signal is unreliable for positioning applications where higher accuracy and robustness are needed. Besides the magnetic field, we also explored a natural static passive electric field, the field between the human body and surroundings, namely the human body capacitance(HBC). HBC is a physiological parameter describing the charge distribution difference between the body and the surroundings and is seldomly explored before. We developed a few wearable, low-cost, low power consumption hardware platforms, either based on an oscillating unit or discrete components composed sensing front end followed by a high resolution analog-to-digital module, to
monitor the variation of the parameter regarding the body movement and environmental variations. Compared with the inertial sensors, the HBC could deliver full-body movement perceiving, meaning that the movement of the legs could be perceived by a wrist-worn HBC sensing unit, which is far beyond the
sensing ability of an inertial sensing unit.
To summarize, we introduced two competitive field sensing modalities for HAR tasks, the magnetic field sensing for position-related services and the passive electric field sensing for full-body action and environmental variation sensing. Both of which were still in an infant stage and not fully explored in the community. The advantages of the two field sensing modalities were demonstrated with a series of position-related and motion-related experiments
Social Distance Monitor with a Wearable Magnetic Field Proximity Sensor
Social distancing and contact/exposure tracing are accepted to be critical strategies in the fight against the COVID-19 epidemic. They are both closely connected to the ability to reliably establish the degree of proximity between people in real-world environments. We proposed, implemented, and evaluated a wearable proximity sensing system based on an oscillating magnetic field that overcomes many of the weaknesses of the current state of the art Bluetooth based proximity detection. In this paper, we first described the underlying physical principle, proposed a protocol for the identification and coordination of the transmitter (which is compatible with the current smartphone-based exposure tracing protocols). Subsequently, the system architecture and implementation were described, finally an elaborate characterization and evaluation of the performance (both in systematic lab experiments and in real-world settings) were performed. Our work demonstrated that the proposed system is much more reliable than the widely-used Bluetooth-based approach, particularly when it comes to distinguishing between distances above and below the 2.0 m threshold due to the magnetic field’s physical properties
The State-of-the-Art Sensing Techniques in Human Activity Recognition: A Survey
Human activity recognition (HAR) has become an intensive research topic in the past decade because of the pervasive user scenarios and the overwhelming development of advanced algorithms and novel sensing approaches. Previous HAR-related sensing surveys were primarily focused on either a specific branch such as wearable sensing and video-based sensing or a full-stack presentation of both sensing and data processing techniques, resulting in weak focus on HAR-related sensing techniques. This work tries to present a thorough, in-depth survey on the state-of-the-art sensing modalities in HAR tasks to supply a solid understanding of the variant sensing principles for younger researchers of the community. First, we categorized the HAR-related sensing modalities into five classes: mechanical kinematic sensing, field-based sensing, wave-based sensing, physiological sensing, and hybrid/others. Specific sensing modalities are then presented in each category, and a thorough description of the sensing tricks and the latest related works were given. We also discussed the strengths and weaknesses of each modality across the categorization so that newcomers could have a better overview of the characteristics of each sensing modality for HAR tasks and choose the proper approaches for their specific application. Finally, we summarized the presented sensing techniques with a comparison concerning selected performance metrics and proposed a few outlooks on the future sensing techniques used for HAR tasks